In technical communications systems, signalling information for the implementation of data communications and/or speech communications between people and/or machines is exchanged between participating equipment. Accordingly, for example, in the mobile-radio sector, signalling messages are exchanged via the air interface between user equipment (UE, “user equipment”, for example, smart phones) and network nodes (for example, base stations).
Before the provision of new equipment, new software releases, new features etc., the error-free message exchange across the relevant interfaces must be tested. Accordingly, functional tests must be implemented, but also system tests, including load and stress tests, regression tests etc. As a rule, these tests are implemented in an automated manner, for example, special test equipment (TE, “Test Equipment”) can exchange messages with the device under test (DUT, “Device under Test”) in the batch mode. An automated test can run over a relatively short time span of, for example, a few minutes or may require a considerably longer time disposed in the region of one hour, several hours or even days.
During the test, the message exchange via one or more interfaces is logged (generally by the TE) and stored in a test log or test protocol. After the completion of the test, the test log can be present in the form of a file, for example, a text file. In general, the log contains a chronological sequence of occurrences or events, that is, the log comprises a list of timestamps, wherein a message is allocated to every timestamp. The data of the message can comprise selected details, for example, given message contents, such as the presence and/or the value of parameters, embedded messages of lower message-log layers and so on.
In general, it is the events listed in the test log and only these which axe available for the identification of problems and determination of their cause.
An evaluation of the test log can be implemented on site in the test laboratory. However, because of its complexity, the evaluation is generally implemented at a later time, and therefore outside the test laboratory. On the one hand, the evaluation comprises the identification of error situations, which includes the recognition that an error situation is present in the first place, and, on the other hand, the search for causes of known error situations. In simple cases, the presence of an error and its cause can be unambiguously inferred on the basis of a message or message sequence. For example, a software error in the DUT can influence a given parameter value of a message; if this is evident from the log, the cause for a restart of a DUT or of a component can be identified.
Often, however, an error situation or an event may, in principle, be attributable to different causes. In such cases, it is difficult to draw an inference about the actually present cause for the error in concrete terms, and this may require a broader evaluation of the log.
A testing of a UE (for example, a smart phone) in which the UE has already successfully synchronised with a mobile-radio network, for instance, on an RLL (“Radio Link Layer”) can be used as an example. It is evident from the test log that the UE suddenly tries to synchronise with the network again; for example, the UE suddenly initiates a new Random-Access-Procedure. The occurrence of this message in fact always indicates that a problem is evidently present here, however, a series of different causes for this behaviour are conceivable; for instance, an error may be present in the UE, but a reaction of the UE to the test conditions may also be involved, such as a simulated power loss in the radio network, for example, as a result of an error in the network or a movement of the UE within the network, for example, in order to bring about a handover etc. It is not easy to determine whether an error situation is present at all in this case, and if it is, to which concrete cause the error can be attributed.
Ambiguous (error) situations, as in the example, represent a fundamental problem in the evaluation of logs from automated tests, for example, in the telecommunications sector.
Searching and evaluation of the log can be implemented by a human tester. However, a test log can also contain very many data even with test durations of only a few minutes, because of the complexity of modern communications systems and the corresponding signalling volume, so that an analysis by human test personnel is very time-consuming. Furthermore, because of the volume of data, it cannot be ascertained that all error situations have been identified as would be desirable. Error situations may possibly also have been be incorrectly evaluated, because only a consequential effect or a subsequent influence of the error is identified rather than the underlying error.
Testing instruments or respectively tools are known which are supposed to support the human tester in the evaluation. For example, parts of the log can be displayed in different colours, in the case of a display on the screen, for instance, in order to improve readability. Accordingly, different message-log levels, for instance, can be displayed differently. Furthermore, a test tool can support the search for a given text string in a text-based test log. The strings then characterise, for example, a given message type, a given error situation etc. Such string searches can also be implemented in an automated manner.
Expert systems for an automated evaluation of test logs are also known, D. Gustaysson, D. Molin, “Expert System for Error Analysis—Rule Based Reasoning Applied on Log Information and Dump Reports from Mobile Phones”, Master's dissertation, June 2010, describes such an expert system. This system comprises knowledge which is represented, for example, in the form of a plurality of rules. The system can accordingly draw inferences, and in fact, specified decision trees are run through dependent upon an evaluation of the rules. A rule comprises a condition component and a result component. A possibly occurring event can be represented in the condition component, and an inference can be represented in the result component. The expert system thus searches the test log for the occurrence of the condition specified in the condition component, which generally corresponds to a string search (“Pattern Matching”), and then draws corresponding inferences from this.
Such decision trees are very suitable for detecting specific errors, that is, well-defined events which are attributable to a previously determinable, unambiguous cause. If a plurality of corresponding decision trees is stored in a data bank, a corresponding plurality of problems in the test log can, in principle, be identified and solved.
Such expert systems therefore represent an efficient tool which supports the human tester in reliably determining the presence of corresponding errors and in determining their cause.
However, the known expert systems cannot efficiently handle error situations which are not clearly defined. Accordingly, problems are not recognised if they do not fit exactly with a previously defined condition. Even the absence of a message means that an expert system can no longer draw a meaningful inference. Known expert systems can therefore support the human tester only in detecting and explaining simple error situations. However, the human tester must generally search through the entire log for the occurrence of complex error situations and clarify these errors exclusively on the basis of personal experience.
In fact, known expert systems can be steadily developed to include an ever-increasing range of conceivable error situations, however, such expansion is subject to limits of scale: for every new error situation, a corresponding new decision tree must be implemented. While constantly improving granularity of the system is required, this leads to a rapidly increasing complexity of the expert system. The maintenance of the system, for example, tracking changes in the message logs, therefore becomes correspondingly elaborate, that is, complex and susceptible to error.
Thus, one object of the present invention is to propose efficient solutions for an automated evaluation of test logs in the field of testing telecommunications equipment, which achieve a reliable support of the human tester in the identification and/or resolution of complex error situations.